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A  contract  between  the  U  .  S  .  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  and  Pennsylvania  State  University, 
The  Milton  S.  Hershey  Medical  Center,  (S.  Streufert,  principal 
investigator)  stipulates  that  the  contractor  will  use  a  simula¬ 
tion  technique  to  determine  whether  complexity  (multidimension¬ 
ality)  of  task  performance  in  complex  managerial  tasks  is 
trainable.  For  that  purpose,  the  contractor  developed  and/or 
improved  a  computer  based  simulation  system  during  the  first  year 
of  effort.  The  simulation  was  to  be  pretested  during  the  second 
year  of  the  contract. 

Initial  development  was  completed  on  Apple  computers  which 
were,  however,  too  limited  in  capacity  to  effectively  operate 
the  simulation  system.  To  overcome  this  problem,  IBM  AT  enhanced 
computers  were  ordered.  Delivery,  as  scheduled  by  IBM,  would 
have  resulted  in  a  three  months  delay  in  scheduled  tasking. 
Unfortunately,  IBM  subsequently  cancelled  all  delivery  dates 
because  of  a  faulty  component  of  the  AT  system  and  rescheduled 
delivery  for  some  8  months  later.  As  a  result,  final  simulation 
development,  design  of  measurement  techniques  and  simulation 
pretesting  was  delayed  for  periods  up  to  12  months.  With  IBM  AT 
systems  finally  available  during  the  late  summer  of  1985, 
efforts  to  finalize  the  simulation  technology  and  to  develop 
measurement  techniques  was  initiated.  Data  collection  on  those 
segments  of  the  contracted  efforts  are  now  completed.  The 
obtained  data  and  the  associated  methodology  will  be  considered 
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in  three  reports. 

The  present  report  is  specifically  concerned  with  measure¬ 
ment.  Previous  simulation  based  measurement  (cf.  Streufert, 
1983)  which  had  included  sixteen  measures  was  extended  to  thirty 
seven  primary  measures  and  twelve  derived  measures.  This  report 
provides  information  on  the  characteristics  and  purpose  of  each 
of  those  measures.  In  addition,  it  will  provide  formulas  or 
related  statements  that  allow  calculation  of  performance  scores 
by  other  researchers  and/or  in  other  settings.  Further,  this 
report  considers  the  Time-Event  Matrix  on  which  measurement  is 
based  . 

A  second,  subsequent  report  will  be  concerned  with  measure¬ 
ment  reliability.  Any  procedure  which  requires  test  and  retest 
of  performance  functions  must  assure  the  reliable  measurement  of 
some  phenomenon  of  interest  on  two  separate  occasions.  For 
present  purposes,  reliability  across  two  measurements  from  two 
parallel  simulations  with  diverse  scenarios  is  required.  With 
training  interspersed,  measurement  across  simulations  must  be 
comparable  if  changes  in  performance  are  to  be  discovered. 
Variable  error  must  be  kept  to  as  low  a  value  as  possible.  The 
second  report  in  this  series  will  focus  on  the  question  of 
inter-simulation  reliability  of  the  measures  that  are  considered 
in  this  report. 

A  third  report  will  be  based  on  data  analysis  that  is  now 
in  progress  and/or  in  the  planning  stage.  This  last  report  in 
the  series  devoted  to  simulation  technology  will  be  concerned 


with  the  interrelationship  among  the  various  measures  (based  on 
factor  analytic  techniques)  and  with  measurement  validity. 
Validity  analysis  will  be  achieved  by  calculating  correlations 
among  measures  that  load  highly  on  obtained  factor  scores  with 
several  secondary  indicators  of  performance,  e.g.  job  level  (at 
age),  number  of  promotions,  income  (at  age)  and  so  forth. 

Reports  that  follow  the  present  series  will  be  concerned  with 
measured  training  effects  per  se . 

Discovering  Performance  Characteristics 

A  few  executives  are  fabulously  successful.  Others  fail 
early  in  their  careers.  Most  do  their  jobs  adequately.  What 
makes  the  difference? 

The  majority  of  executives,  especially  at  senior  levels, 
are  highly  intelligent.  They  have  gained  ample  experience. 

They  have  been  well  trained.  Yet,  if  we  observe  what  they 
understand,  what  they  know  and  what  decisions  they  make  we  still 
cannot  distinguish  those  who  are  great  from  those  who  are 
mediocre.  Many  observers  of  executive  performance  have  watched 
and  recorded  what  great  executives  do.  Unfortunately,  their 
descriptions  of  excellence  would  hold  in  one  organizational 
setting  but  may  not  apply  in  another. 

The  difference  between  excellence  and  mediocrity  is  not 
generally  found  in  what  an  executive  may  know,  think  or  decide. 
The  difference  is  not  just  determined  by  the  choice  of  correct 
and  appropriate  actions.  It  is  not  based  on  the  capacity  to 


follow  a  preconceived  strategy  toward  some  fixed  goal.  It  is 


not  found  in  the  tendency  to  amass  all  relevant  information  that 
can  be  obtained.  Even  mediocre  executives  can  do  all  those 
things  -  and  most  can  probably  do  them  well. 

Excellence  in  executive  performance  depends  on  how  a n 
experienced  executive  thinks,  plans  and  responds  to  task  demands 
It  is  inherent  in  the  creative  alternatives  that  are  considered. 
It  is  evident  in  the  development  of  multiple  strategies  that 
proceed  from  step  to  step,  always  adaptive,  with  goals  that  are 
general  and  become  more  and  more  defined  as  they  are  approached. 
It  is  found  in  conclusions  that  are  checked,  reconsidered  and 
reassessed  as  more  information  arrives.  It  is  found  in  the 
adaptive  flexibility  that  permits  different  approaches  to 
different  people  and  tasks,  that  permits  the  executive  to  switch 
from  strategic  planning  to  immediate  action  when  emergencies 
arise  and  to  return  to  a  planning  mode  when  the  emergency  has 
been  conquered. 

How  can  one  measure  that  executive  excellence?  It  requires 
a  view  of  an  executive's  actions  over  time,  in  different  situa¬ 
tions,  relevant  to  diverse  task  demands.  It  requires  assessing 
how  executives  develop,  interrelate  and  apply  their  ideas.  It 
requires  an  analysis  of  the  steps  which  executives  take  to 
approach  and  attain  their  goals.  A  simple  test,  an  interview  or 
even  a  short  task  cannot  provide  us  with  adequate  information 
about  executive  capacity.  Just  as  a  photographic  snapshot 
cannot  tell  us  what  a  person  will  do  in  a  few  hours,  the  usual 
measurement  techniques  and  well  intentioned  observations  tell  us 


little  about  executive  performance  across  a  variety  of  task 
demands . 

There  have  been  very  few  efforts  that  have  analyzed  execu¬ 
tive  performance  through  a  number  of  potentially  interrelated 
events  or  through  a  series  of  decisions  that  are  part  of  a 
strategic  effort.  If  we  wish  to  discover  h ow  executives  func¬ 
tion,  such  a  time  oriented  analysis  is  exactly  what  is  needed. 
Observations  that  are  more  or  less  casual  (e.g.  those  of  Peters 
and  Waterman,  1982,  Peters  and  many  other  recent  observers  of 
executive  functioning)  are  indeed  valuable,  but  they  rarely 
generate  the  necessary  depth  of  understanding.  Much  more  useful 
are,  for  example,  the  careful  efforts  of  Dan  Isenberg  (1984)  who 
has  recorded  both  the  actions  and  the  rational  for  consecutive 
executive  actions  across  considerable  time  periods.  The  present 
approach  is  quite  similar  to  that  of  Isenberg,  but  uses  a 
simulation  methodology  to  (1)  automate  data  collection  proce¬ 
dures,  (2)  to  obtain  multiple  independent  measures  of  perform¬ 
ance,  (3)  to  maintain  control  over  relevant  independent  variable 
manipulation  across  time  and  participants  and,  last  not  least, 
(4)  to  explore  lengthy  periods  of  performance  during  which 
strategic  efforts  might  develop.  The  simulation  permits  this 
kind  of  performance  measurement  in  reduced  time  periods  via  the 
compression  of  time  experience.  The  technique  is  described  in 
the  next  section  of  this  report. 

Data  Collection  via  a  Qua s i -Expe r i men t a  1  Simulation 


present  research  effort  is  best  defined  as  a  "quasi  experimental 
simulation"  (Fromkin  and  Streufert,  1983;  Streufert  and  Swezev, 
1985).  The  methodology  allows  the  introduction  of  controlled 
independent  variables  not  only  at  the  beginning  of  the  simulation 
task  but  maintains  control  over  independent  variables  throughout 
the  simulation.  The  considerable  length  of  time  (several  hours) 
during  which  participants  deal  with  their  simulated  environment 
permits  the  simultaneous  and/or  successive  manipulation  of 
several  experimental  variables. 

Continued  control  over  experimental  variables  throughout 
the  simulation  permits  the  application  of  two  kinds  of  perform¬ 
ance  measurement:  (1)  repeated  collection  of  data  throughout  the 

simulation  period  to  record  the  effects  of  conditions  that  are 
experimentally  modified  as  the  simulation  progresses  (for 
example  one  might  measure  the  effects  of  different  load  levels, 
e . g . ,  Streufert  and  Driver,  1965;  Streufert  and  Schroder,  1965; 
Streufert,  1970),  and  (2)  one-time  collection  of  data  that  are 
relevant  to  some  specific  preprogrammed  single  event  which 
occurs  in  the  simulation.  The  former  measurement  techniques  are 
particularly  useful  to  assess  the  (structural)  "style  of  think¬ 
ing"  that  underlies  task  performance.  The  latter  can  capture 
structural  thought  processes  only  to  some  extent,  but  is  espe¬ 
cially  useful  to  obtain  information  about  the  appropriateness, 
speed  and  accuracy  of  responses  to  specific  task  demands. 

The  development  of  measurement  in  our  quasi-experimental 
simulation  technique  is  best  described  as  an  ongoing  process. 


Since  the  computer  system  which  operates  the  simulation  technol¬ 
ogy  records  a  large  variety  of  actions,  plans  and  response 
characteristics  of  participants,  newly  developed  measures  can 
often  be  derived  from  data  collected  at  a  much  earlier  date. 

Mew  measures  may  or  may  not  be  entirely  independent  of  previous 
measures,  (overlap  of  meaning),  others  may  be  entirely  indepen¬ 
dent  (orthogonal).  The  precise  interrelationships  of  the  various 
measures  will  be  evaluated  in  a  factor  analytic  procedure  based 
on  data  from  approximately  80  simulations.  The  results  will  be 
reported  some  time  later  this  year  (likely  in  the  summer  of 
1986)  . 

At  present,  the  computerized  simulation  system  records 
performance  characteristics  in  terms  of  interrelationship 
vectors  and  calculates  scores  for  more  than  fifty  performance 
measures.  Thirty -seven  of  those  measures  are  more  or  less 
complex  calculations  of  performance  frequencies.  The  remaining 
measures  relate  these  frequencies  to  the  number  of  actions  taken 
(decisions  made)  to  obtain  proportions  of  performance  character¬ 
istics  as  they  relate  to  general  activity. 

Data  collection  for  the  simulation  technique  is  reflected 
in  a  graphic  procedure  that  has  become  known  as  a  "Time-Event 
Matrix."  The  rationale  for  data  collection  and  measurement 
procedures  are  best  explained  via  that  matrix  system.  The 
following  section  of  this  paper  will  focus  on  the  matrix  proced¬ 
ure.  Subsequently,  the  measures  themselves  will  be  explained  in 


some  detail. 
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The  Time-Event  Matrix 

The  task  of  an  individual  or  a  task-oriented  group  operating 
in  the  world  outside  of  the  simulation  laboratory  is  rarely 
limited  to  deciding  on  a  single  event  within  a  limited  context. 
For  example,  most  decision  makers  in  applied  settings  must 
respond  to  an  ongoing  series  of  inputs  from  their  environment. 

The  resulting  output  is  usually  a  sequence  of  actions  determined 
in  part  bv  some  plan  and  in  another  part  by  the  necessities  of 
dealing  with  current  events.  The  output  may  consist  of  primaril 
"respondent"  actions  or  it  may  reflect  some  degree  of  "strategy," 
i.e.,  decisions  which  are  interrelated  with  each  other  and  occur 
in  a  planned  sequence  to  achieve  some  kind  of  goal.  Whether 
or  not  individual  or  group  actions  do  reflect  pure  respondent 
behavior,  whether  they  reflect  some  kind  of  strategy  (and  the 
level  and -'or  characteristics  of  that  strategy)  may  be  of  consid¬ 
erable  importance  for  the  outcome  of  the  task  effort.  The 
majority  of  previous  researchers  have  not  focused  on  measuring 
or  describing  such  differences.  To  alleviate  that  problem, 
Streufert  and  associates  have  developed  a  time-event  matrix  to 
help  researchers  or  observers  to  identify  different  kinds  of 
actions  and  their  frequencies,  as  they  occur  in  naturally 
complex  task  settings.  Reliability  and  validity  for  some  of  the 
measures  derived  from  time-event  matrices  have  been  established 
in  previous  efforts.  Additional  reliability  and  validity  data 
will  be  reported  in  a  subsequent  report. 

Performance  quality,  particularly  in  complex  tasks,  is 


determined  by  at  least  two  major  components  of  individual  or 
group  efforts:  (1)  appropriate  knowledge  about  what  responses 
are  potentially  correct  or  incorrect  (where  possible)  and  (2) 
the  ability  to  develop  plans,  and  to  respond  at  the  right  time 
with  optimal  combination  of  responses,  i.e.,  the  use  of  strategy 
The  time-event  matrix  was  designed  to  measure  the  latter  of  the 
two  components.  In  many  cases,  the  first  component,  i.e., 
appropriate  content  knowledge  and  understanding  of  the  task 
situation  can  be  assumed,  as  long  as  sufficient  training  and 
experience  is  available.  Nonetheless,  specific  quality  perform¬ 
ance  measures  are  collected  as  well,  even  though  they  may  not  be 
directly  captured  by  the  time-event  matrix  procedure. 

Time-event  matrices  can  be  used  to  measure  a  variety  of 
task  performance  activities,  depending  on  the  interests  and 
orientations  of  the  researcher  or  observer.  This  paper  cannot 
cover  all  of  the  purposes  for  which  the  matrix  can  or  has  been 
employed.  For  greater  ease  of  communication,  let  us  focus  on 
decision-making  matrices  as  an  example  for  all  matrix  possibili¬ 
ties.  It  should  be  remembered,  however,  that  most  other  perform 
ance  areas,  aside  from  decision-making,  could  have  been  selected 
equally  well. 

The  time-event  matrix  technique  was  initially  developed 
to  measure  the  interrelationships  among  actions  across  time  and 
the  effects  of  information  flow  which  precedes  those  actions. 
Details  about  the  construction  of  time-event  matrices  will  be 
discussed  below.  At  this  point  it  is  merely  important  to  be 


aware  that  these  matrices  capture  all  data  about  incoming 
information,  about  decisions  and  other  actions  based  on  that 
information,  about  interrelationships  among  information  and 
decisions  as  well  as  interrelationships  among  decisions  (e.g., 
strategy).  The  matrices  may  be  used  to  collect  data  on  measures 
which  reflect  how  task  oriented  individuals  or  groups  process 
information  and  how  that  information  processing  determines  or 
affects  observed  performance.  Measures  based  on  the  time-event 
matrices  may  be  considered  "intermediate"  assessments  of  perform¬ 
ance  quality.  They  provide  a  necessary  vehicle  for  estimating 
and  defining  important  action  antecedents  of  performance  (criter¬ 
ion  based)  quality,  particularly  in  complex  tasks.  Their 
predictive  validity  for  success  (e.g.,  managerial  excellence)  may 
be  accessed  in  research  designs  that  obtain  the  predictive 
validity  would  hamper  each  measure  (to  be  reported  subsequently). 

Tasks  and  their  requirements  differ.  The  same  strategy  is 
not  necessarily  useful  in  all  task  settings  (aside  from  differ¬ 
ences  in  knowledge  content).  As  a  result,  the  measures  derived 
from  the  time-event  matrix  should  be  carefully  validated  against 
each  general  performance  task.  Many  tasks  will,  of  course, 
produce  quite  similar  patterns  of  "optimal"  measurement  levels 
to  criterion.  As  a  matter  of  fact,  a  number  of  validations  have 
shown  that  specific  score  levels  for  the  measures  that  will  be 
presented  later  tend  to  be  quite  robust  across  a  number  of  tasks 
and  a  number  of  performance  environments.  Where  adequate 
training  and/or  sufficient  experience  is  likely  to  result  in  few 
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(if  any)  content  errors  in  performance,  predictions  of  quality 
task  performance  made  on  the  basis  of  matrix  measurement  is  of 
substantial  value.  Where  training,  task  familiarity,  knowledge 
and  experience  with  task  requirements  is  minimal,  the  scores 
would  provide  information  about  the  capacity  to  acquire  excellen¬ 
ce  after  training. 

The  dimensions  of  a  matrix  are  time  and  action  (here 
decision)  types.  Each  will  be  discussed  in  turn: 

( 1 )  Time 

Time  in  the  matrix  is  plotted  horizontally.  There  are  no 
particular  restrictions  on  the  gradations  to  be  used  (no  matter 
whether  time  proceeds  normally  or  is  -  as  in  some  simulations  - 
expanded  or  condensed),  except  that  events  which  occur  sequen¬ 
tially  and  independently  of  each  other  must  appear  on  different 
time  points.  The  time  dimension  moves  from  the  left  to  the 
right.  The  units  of  the  scale  used  are  not  of  significance, 
except  that  decision-making  sequences  which  are  to  be  compared 
must  contain  the  same  scale  units  (since  the  formulas  should 
calculate  comparable  values). 

( 2 )  Decision  Categories 

Decision-making  tasks  and  settings  differ.  Consequently 
decisions  employed  differ  as  well.  For  example,  executives 
dealing  with  the  potential  purchase  of  another  corporation  may 
be  concerned  with  such  action  areas  as  establishing  the  value  of 
the  other  company,  determining  potential  duplication  of  effort, 
etc.  On  the  other  hand,  military  decision  makers  may  be 


concerned  with  troop  movements,  air  support  decisions  and  so 
forth.  In  other  words,  groupings  of  decisions  (decision  categor 
ies)  must  be  established  separately  for  each  general  group  of 
decision-making  situations.  Selection  of  predefined  decision 
categories  is  best  accomplished  by  experts  in  the  field.  The 
selected  categories  should  be  inclusive,  where  possible  of 
approximately  equal  breadth,  and  conceptually  meaningful  and 
consistent.  They  should  clearly  differ  from  each  other  in 
activity,  method,  meaning,  etc.  Decision  categories  should 
provide  the  potential  for  use  by  decision  makers.  While  some 
decision  makers  would  likely  use  one  group  of  decisions,  others 
may  focus  on  a  different  group,  of  course  with  considerable 
overlap  . 

While  there  is  no  restriction  on  the  number  of  potential 
decision  categories  that  might  be  represented  in  a  time-event 
matrix,  the  inclusiveness  of  decision  categories  should  be 
selected  so  that  decision  makers  utilize,  on  the  average, 
somewhere  between  ten  and  fifty  different  categories  of 
decisions^  in  any  time  sequence  that  lasts  for  several  hours. 
Note,  however,  that  these  suggestions  are  ideal  requirements  and 
do  not  supercede  the  practical  characteristics  of  any  particular 
task  situation.  For  example,  if  a  decision  task  requires  only 
one  kind  of  decision,  one  cannot  "manufacture"  other  decision 
categories  by  hook  or  crook.  In  effect,  the  use  of  the  decision 

^Since  decision  makers  would  rarely  employ  all  available 
decision  categories,  the  potential  for  considerably  more  than  50 
categories  may  be  provided. 


matrix  in  such  simple  situations  would  have  little  value.  For 
example,  if  all  actions  reflected  troop  movements,  then  splitting 
decisions  by  the  unit  moved  may  not  be  useful. 

( 3 )  Decision  Points 

Once  time  is  plotted  horizontally  and  decision  categories 
(as  selected,  for  example,  by  an  expert  panel)  are  plotted 
vertically,  each  decision  made  by  an  individual  or  a  group  of 
decision  makers  (as  desired  by  the  researcher  or  observer)  can 
be  represented  by  a  point  placed  vertically  beneath  the  time 
when  that  decision  was  made  (or  announced,  or  transmitted,  again 
depending  on  the  intent  of  the  researcher  or  observer)  and 
horizontally  next  to  the  decision  category  represented.  All 
decisions  can  thus  be  placed  in  the  matrix. 


Information  input  that  is  received  during  the  simulation  (a 
potential  carrier  for  independent  variable  manipulations),  is 
considered  as  it  relates  to  decision  output  (this  limitation  was 
chosen  for  convenience  and  is  not  necessary).  Any  unit  of  input 
which  leads  to  an  output  is  marked  (e.g.,  by  a  *)  under  its 
appropriate  (input)  time  and  in  front  of  (on  the  same 
dec i s i on- t y pe  line  as)  any  decision  made  as  a  consequence  of  that 
input.  The  input  is  placed  in  advance  of  each  output  which  it 
produced,  i.e.,  the  same  input  may  occur  on  more  than  one 
horizontal  ( dec i s i on -c a t e go r v  )  line.  The  distance  on  the 
horizontal  between  the  input  *  and  the  decision  point  is  marked 
with  a  dashed  line.  It  reflects  the  time  elapsed  between  receipt 


of  information  and  relevant  response  .  - 


( 5 )  Diagonals 

As  stated  above,  we  are  interested  in  the  relationships 
among  decisions  as  they  reflect,  for  example,  the  development  of 
plans  or  strategies.  Consequently  we  wish  to  know  whether  a 
decision  made  at  one  time  is  related  (leads)  to  a  decision  at  a 
later  time.  Where  a  decision  in  one  category  is  made  to  make  a 
later  decision  representing  another  category  possible,  the  two 
decisions  are  connected  across  time  with  a  diagonal  line.  An 
arrow-head  points  forward  toward  the  later  decisions . ^  If  two 
decisions  show  an  isolated  relationship  to  each  other,  a  single 
arrow  is  drawn.  If,  on  the  other  hand,  the  decision  maker(s) 
decide(s)  to  engage  in  decision  types  A  and  B  at  time  one  to 
allow  for  action  C  in  the  future,  and  wants  to  accomplish  C  to 
allow  D  to  occur  even  later,  and  if  all  these  decisions  are 
actually  made  in  time,  a  longer  chain  of  diagonal  connections 
is  established: 


2  S  e  e  the  section  on  Integration  Time  Weight  (below)  for  a 
discussion  of  time  measurement. 

^Such  diagonal  connections  in  the  matrix  will  later  be 
referred  to  as  "integrations." 
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( 6 )  End  Effects 


Whether  or  not  a  diagonal  is  drawn  depends,  of  course,  on 
whether  a  planned  later  decision  is  indeed  produced  as  a  follow¬ 
up  to  an  earlier  decision.  Where  a  decision  task  ends  abruptly, 
the  opportunity  to  carry  out  planned  subsequent  decisions  may 
not  exist.  Ending  a  simulation  could  arbitrarily  limit  the 
number  of  diagonals  produced  by  decision  makers  as  reflected  in 
obtained  measures  (see  below).  Randomization  experimental 
events  may,  in  some  cases,  be  employed  to  avoid  a  constant 
error.  Alternatively,  the  probabilities  of  scores  that  should 
have  been  obtained  during  latter  time  periods  in  the  simulation 
may  be  calculated  by  comparing  earlier  and  late  performance 
characteristics.  This  latter  method  is  especially  useful  if  a 
fixed  order  of  simulated  events  is  needed  for  purposes  of  data 
collection. 

( 7  )  Establishing  Relationships  in  the  Matrix 

It  is  important  to  establish  clear  relationships  (a) 
between  inputs  and  subsequent  output  decisions,  and  (b)  among 
decisions  which  are  causally  or  strategically  related  (represent 
ed  by  diagonals).  Of  course,  the  only  perfect  representation 
of  these  interrelationships  exists  in  the  brain  of  the  decision 
maker(s)  at  the  moment  relevant  decisions  are  made.  Any  measure 
of  those  relationships  can,  consequently,  be  subject  to  some 
error.  Clearly,  it  is  important  to  opt  for  the  least  amount 
of  error  in  any  experimental  or  observational  setting.  Certain¬ 
ly,  the  error  levels  of  obtained  data  would  likely  be  much 


smaller  in  a  well  designed  complex  experimental  simulation  than 
in  research  based  on  observation  in  an  ongoing  free  environment. 
For  example,  in  experimental  simulations,  records  of  planning 
can  be  obtained  directly  from  participants  during  the  planning 
process.  In  real-world  task  environments,  less  precise  tech¬ 
niques  such  as  post-hoc  interviews  are  required. 

Ideally,  decision  maker(s)  should  be  asked  immediately 
(upon  making  a  decision)  to  indicate  (a)  any  information  received 
upon  which  the  decision  is  based,  and  (b)  any  planned  subsequent 
decisions  that  they  might  employ  as  a  follow-up  to  a  current 
decision.  That  kind  of  data  can  be  obtained  in  complex  experi¬ 
mental  simulations  (the  participants  may  have  to  be  persuaded, 
however,  that  indicating  previously  received  information  and 
indicating  planned  future  decisions  would  be  of  value  to  them  in 
terms  of  long  range  outcome).  In  many  free  simulations  (partic¬ 
ularly  if  interrupt  control  is  lacking)  or  in  the  observation  of 
real-world  decision-making  environments  these  kinds  of  questions 
cannot  be  asked.  Collecting  those  data  after  completion  of  their 
task  often  introduces  serious  bias.  One  might  require  experts  to 
consider  all  decisions  that  were  made  and  to  judge  whether  these 
decisions  represent  responses  to  previous  information  and/or 
were  part  of  decision-making  sequences  that  should  be  represented 
by  diagonal  interconnections.  Hopefully,  interjudge  reliabil¬ 
ity  for  such  a  task  would  be  high.  Previous  experience  has  shown 
that  judges  produce  little  variable  error  in  making  these 
judgments.  As  long  as  the  judges  have  no  particular  biases  for 
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or  against  certain  decision  makers  they  are  evaluating,  constant 
errors  across  various  samples  may  not  be  excessive.  Nonetheless, 
the  immediate  data  collection  method  that  can  be  employed  in 
experimental  simulations  has  considerable  advantages  over  other 
procedures  . 

Establishing  connections  between  inputs  and  decisions  on 
the  basis  of  expert  judgments  is  relatively  easy.  Respondent 
decisions  are  typically  directly  related  to  the  verbal  content 
of  input  information  or  describe  the  same  location  or  information 
source  contained  in  the  input.  When  such  commonalities  are 
seen,  a  connection  may  be  assumed  to  exist.  More  difficult  is 
the  interpretation  of  connections  among  decisions.  Obviously, 
where  one  decision  directly  refers  to  a  previous  decision 
("Order  tne  unit  which  we  moved  to  quadrant  X5  to  fire  on...")  a 
diagonal  connection  is  directly  established.  However,  is  this  a 
forward  or  a  backward  diagonal?  If  we  were  able  to  ask  the 
decision  maker(s)  about  future  decisions,  when  the  original 
decision  to  move  the  unit  was  made,  then  we  would  now  know.  If 
we  were  not  able  to  ask  (in  free  simulations  of  the  type  describ¬ 
ed  above  or  in  real-world  applied  decision-making  settings), 
then  we  cannot  be  certain.  In  these  cases  forward  and  backward 
diagonals  cannot  be  distinguished  and  arrowheads  cannot  be 
drawn. 

Whenever  no  clear  relationship  is  stated  by  the  decision 
maker(s),  aids  must  be  used  to  determine  whether  relationships 
may  exist.  Such  commonalities  among  decisions  as  addressee, 
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location,  action  etc,  can  be  useful  for  that  purpose.  The  most 
reliable  indicator  of  interrelationship  is  probably  location. 

In  a  military  setting,  to  give  a  relatively  simple  example, 
moving  artillery  to  quadrant  X5,  asking  it  to  fire  on  Y  4  ,  moving 
infantry  to  Y5  and  finally  ordering  the  infantry  to  attempt  to 
take  Y4  would  reflect  a  minimal  series  of  interrelated  decisions 
across  time . 

It  should  be  noted  here  that  moving  troops  to  Y5  and 
another  troop  unit  (both  infantry)  to  X5  (at  a  later  time)  would 
not  result  in  a  diagonal  connection:  both  actions  are  included 
within  the  same  decision  category:  repetitious  action  is  not 
necessarily  representative  of  strategic  action.  If,  on  the 
other  hand,  both  units  are  later  asked  to  attack  Y4,  separate 
diagonal  connections  between  each  of  the  two  movements  and  the 
later  attack  would  be  drawn. 

Some  decision  sequences  may  be  difficult  to  judge  in  terms 
of  their  potential  interconnectedness.  To  the  degree  to  which 
the  judge  can  develop  a  picture  of  the  strategy  decision  makers 
used  (or  if  the  judge  can  obtain  advance  information  about  their 
plans),  the  determination  of  strategic  relationships  will  be 
considerably  easier.  In  any  case,  if,  after  considerable 
thought,  a  judge  is  uncertain  whether  two  decisions  are  or  are 
not  related  to  each  other,  it  is  better  to  err  by  omission. 
Uncertain  relationships  (interconnections)  should  not  be  scored 
to  avoid  artificially  inflating  some  of  the  measures  (below) 
which  can  show  near  quadratic  effects  of  erroneously  scored 


relationships.  Of  course,  the  use  of  direct  questioning  that  is 
possible  in  experimental  and  q ua s i -e x pe r i me n t a  1  simulation 
techniques  would  avoid  such  quandary  altogether. 

An  example  of  two  decision  matrices  is  provided  on  page  21. 
The  figure  shows  decision  matrices  generated  by  two  groups  of 
participants  which  differed  in  their  decision-making  styles 
(complexity)  . 

MEASURES 

A  considerable  number  of  measures  are  presented  and  explain¬ 
ed  in  this  report.  Additional  measures  can  be  developed  if  they 
are  useful  for  any  specific  task  at  hand.  Calculation  of  the 
measures  assumes  either  that  a  time-event  matrix  has  been  drawn 
(by  computer,  if  obtained  from  an  experimental  simulation,  or  by 
judges,  if  obtained  from  a  free  simulation  or  a  real-world 
decision-making  environment)  or  that  computer  generated  vectors 
for  task  performance  events  are  available.  The  various  measures 
were  developed  to  reflect  different  kinds  of  task  performance. 
In-and-of-themselves,  each  measure  cannot  be  considered  a 
reflection  of  "good"  versus  "bad"  performance  with  regard  to  any 
particular  criterion  without  considering  momentary  demands  (e.g., 
environmental  conditions).  Without  question,  there  are  situ¬ 
ations  where  complex  strategic  planning  (as,  for  example, 
reflected  in  the  QIS  and  WQIS  measures,  below)  is  of  considerable 
value.  On  the  other  hand,  there  are  situations  where  such 
planning  would  be  superfluous  and  inappropriate  since  task 
demands  may  require  immediate  (e.g.,  respondent,  see  below) 
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actions.  Measures  for  both  kinds  of  these  activities  (and  many 
relevant  others)  are  included.  Each  measure  and  its  purpose  will 
be  discussed  below.14  Where  necessary,  calculation  examples  will 
be  provided. 

For  convenience  of  communication,  we  will  again  focus  on 
time-event  matrices  involving  decision  making.  The  measures 
will  be  presented  in  the  order  in  which  they  are  printed  out  by 
the  computer  program,  i.e.,  the  order  in  which  they  were  develop¬ 
ed.  Note,  that  similar  measures  may  not  be  located  adjacently 
to  each  other. 

1.  Number  of  Decisions 

This  measure  reflects  the  amount  of  decision  making  activ¬ 
ity.  It  consists  of  a  count  of  the  number  of  decisions  made, 
i.e.,  the  number  of  points  in  the  matrix  during  any  period  of 
time  that  is  of  interest.  The  formula  for  this  measure  may  be 
written  as 


where  1  through  p  indicates  the  time  period(s)  of  interest  and  d 
is  a  decision  made  within  that  time. 

2.  Number  of  Respondent  Decisions 

This  measure  indicates  the  number  of  information  items  that 

^Some  measures,  as  the  reader  will  note,  are  based  on 
factorial  combinations  of  specific  values.  To  reduce  artifactual 
spread  in  the  distribution  of  higher  values  (increased  artificial 
distances  from  regression  lines)  some  measures  are  calculated  via 
natural  logarithm  transformations. 


were  utilized  in  responses  to  environmental  information.  Since 
any  one  respondent  decision  may  be  responsive  to  multiple 
information,  the  score  for  this  measure  may  exceed  the  score  for 
number  of  decisions. 


where 


r  j  is  any  response  to  information  within  a  given  time 
period  (see  below)  after  receipt  of  relevant  information.  A 
response  (respondent  decision)  in  this  measure  is  counted 
more  than  once  if  it  responds  to  two  or  more  items  of 
relevant  information  that  was  received  previously. 

Whether  a  decision  responds  to  previously  received  informa¬ 
tion  should  ideally  (as  discussed  earlier)  be  determined  by  a 
verbal  indicator  from  the  decision  maker(s).  That  is  the  case 
in  all  experimental  or  quasi  experimental  simulations.  For  other 
research  settings,  it  may  have  to  be  determined  by  competent 
judges  as  long  as  appropriate  access  to  the  decision  maker(s) 
for  questioning  is  not  available. 

Different  decision-making  situations  require  diverse  time 
frames  for  the  processing  and  a c c e s s i ng /c ommuni c a t i ng  of  informa¬ 
tion  and  development  of  subsequent  decisions.  Respondent 
decisions  (as  defined  here,  see  some  potential  modifications 
below)  are  most  often  made  quickly  after  receipt  of  incoming 
information.  The  decision  output  is  usually  not  extensively 
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pondered  and  is  rarely  considered  in  terms  of  existing  or 
emerging  plans  (strategy).  For  example,  a  respondent  decision 
to  the  intrusion  of  enemy  aircraft  into  friendly  airspace  may 
involve  immediate  defensive  action.  Certainly  the  reasons  for 
that  intrusion  may  be  considered  subsequently  and  may  be  reflect¬ 
ed  in  future  activities  that  may  or  may  not  be  strategic  (reflect 
planning).  Nonetheless  the  initial  action,  occurring  as  immedi¬ 
ately  as  possible,  represents  (often  quite  appropriate)  one-to- 
one  responding  to  information.  In  situations  where  any  meaning¬ 
ful  response  to  information  must  occur  quickly,  a  time  restric¬ 
tion  for  scoring  actions  as  responsive  might  be  introduced.  The 
time  limitation  between  receipt  of  information  and  a  response 
which  determines  whether  a  decision  follows  receipt  of  that 
information  with  sufficient  rapidity  to  qualify  as  a  respondent 
decision  must  depend  on  the  constraints  of  the  decision-making 
situation.  In  other  words,  that  time  frame  must  be  determined 
individually  for  each  group  of  decision-making  settings  (or 
scenarios)  that  are  of  interest.  However,  that  time  frame 
cannot  subsequently  be  changed  from  one  person  or  group  to  the 
next,  if  meaningful  comparisons  are  to  be  made. 

It  should  be  noted  that  two  variants  of  the  respondent 
decision-making  measure  have  been  used  with  success.  One  is 
"retaliatory  decision  making"  (cf.  Measure  8).  In  this  measure, 
responses  to  information  receipt  are  not  included  in  the  value  of 
"  r "  i_f  the  relevant  decision  is  part  of  an  integrated  decision 
sequence  (i.e.,  is  counted  in  the  Number  of  Integrations 


measure).  The  assessment  of  retaliatory  decrsion  making  provides 
an  estimate  of  non-strategic  respondent  behavior.  A  second 
modification  of  the  respondent  decision-making  measure  eliminates 
any  time  constraints  for  the  information-decision  sequence.  Here 
all  decisions  made  in  response  to  information  (no  matter  how  much 
delayed)  are  counted.  The  resulting  score  reflects  the  total 
amount  of  respondent  activity.  Note,  however,  that  these 
modifications  of  the  respondent  decision-making  measure  are  not 
statistically  independent  (orthogonal)  of  each  other.  Nonethe¬ 
less  they  can  be  quite  useful  for  specific  research  or  observa¬ 
tion  intents . 

3.  Number  of  Decision  Categories 

This  measure  is  a  simple  count  of  the  number  of  decision 
categories  decision  makers  use  during  any  specific  time  period. 

It  may  be  conceived  as  an  approximate  estimate  of  the  degree  of 
differentiation  in  decision  making.  Any  category  which  is  part 
of  the  count  may  have  been  used  once  or  more  than  once.  The 
measure  reflects  whether  the  decision  maker(s)  would  be  likely 
to  select  smaller  or  larger  numbers  of  action  types.  In  addi¬ 
tion,  further  analysis  could  reveal  whether  decision  maker(s) 
are  likely  to  select  certain  specific  actions  and  eliminate 
others  from  consideration.  The  basic  measure  may  be  written  as 
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where  C  is  the  number  of  categories  employed  and  1  through  p  is 


the  time  period  of  participation  in  the  simulation  that  is  of 
interest  for  analysis  and  interpretation. 

4.  (and  7).  Number  of  Integrations  (Forward,  Backward  or  Total) 


The  score  for  Number  of  Integrations  reflects  the  frequen¬ 
cy  of  the  application  of  single  steps  of  strategy. 
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or 


(  i  b  ) 


or 


( ib  +  if  ) 


where 


if  are  forward  integrations  (relationships,  i.e.,  connec¬ 
tions  among  decision-making  points  with  diagonal 
arrows  pointing  forward),  (Measure  4), 
if,  are  backward  integrations  (relationships,  i.e.,  connec¬ 
tions  among  decision-making  points  with  diagonal  arrows 
pointing  backward),  (Measure  7), 
i  are  integrations,  i.e.,  relationships  where  direction¬ 
ality  cannot  be  established,  (Measure,  4,  modified). 

As  discussed  earlier,  some  decision-making  tasks  (particu¬ 
larly  real-world  decision-making  settings  where  the  researcher 
or  observer  cannot  interfere)  may  not  lend  themselves  to  ques¬ 
tioning  the  decision  maker(s)  about  the  intent  of  decisions. 
Consequently,  it  may  be  impossible  to  determine  whether  a 
connection  (relationship)  among  decisions  reflects  forward 
integrations  (planning  a  later  decision  at  the  time  an  earlier 
decision  has  been  made),  or  reflects  backward  integration, 

(using  a  previous  decision  to  advantage  although  the  connection 
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was  not  considered  at  the  time  the  earlier  decision  did  occur). 
However,  whenever  possible,  forward  diagonals  in  the  matrix 
should  be  counted  as  forward  integrations  and  backward  diagonals 
should  be  counted  as  backward  integrations.  Translation  of 
diagonals  into  integration  scores  is  achieved  on  a  one-to-one 
basis:  Counting  the  number  of  diagonals  of  a  specific  type 

produces  the  relevant  integration  score.  Where  no  distinction 
between  forward  and  backward  diagonals  can  be  made,  integrations 
are  counted  without  concern  for  the  direction  of  arrowheads. 
Example 

For  simplicity's  sake,  let  us  return  to  the  example  matrix 
in  Figure  2,  page  21.  The  upper  matrix  contains  two  forward 
diagonals,  i.e.,  a  score  of  2  for  if  (forward  integrations).  It 
contains  three  backward  diagonals,  i.e.,  a  score  of  3  for  i  5 
(backward  integrations).  The  score  for  i  (if  +  if,)  would  be  3. 
Obviously  the  score  for  the  lower  matrix  in  Figure  2  is  consider¬ 
ably  higher. 

The  measures  concerned  with  number  of  integrations,  integra¬ 
tion  time  weight  and  QIS  depend  on  the  diagonals  connecting 
earlier  and  subsequent  decisions,  indicating  that  an  earlier 
decision  made  a  later  decision  possible  (strategic  time 
sequence).  Where  an  entire  matrix  is  analyzed,  simple  counting 
or  statistical  processing  of  the  number  of  diagonals  is  suffi¬ 
cient.  However,  if  an  experimenter  or  observer  is  concerned 
with  a  limited  time  period  as  part  of  a  larger  decision  time 
sequence  (e.g.,  if  different  experimental  conditions  are 
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introduced  into  an  experimental  simulation  or  if  artificial  or 
natural  probes  are  utilized  in  a  free  simulation),  diagonals 
will  often  cross  the  time  lines  that  describe  a  period  of 
interest.  In  that  case,  diagonals  are  scored  for  the  time 
period  during  which  they  originate.  If  distinctions  between 
backward  and  forward  integrations  (diagonals  with  backward  and 
forward  arrows)  can  be  made,  then  backward  integrations  will  be 
credited  to  the  period  of  the  subsequent  (of  two)  decisions. 
Forward  integrations  will  be  credited  to  the  period  of  the 
initial  (of  two  interconnected)  decisions.  If  no  distinctions 
between  forward  and  backward  diagonals  can  be  made,  all  diagonals 
are  credited  to  the  time  period  of  the  initial  decision.  If 
specific  experimental  manipulations  are  utilized  in  experimental 
or  quasi-exper imental  simulations,  the  manipulations  (if  manipu¬ 
lated  "within"  across  time)  should,  where  possible,  be  randomized 
in  order  to  replace  constant  error  with  variable  error.  Adjust¬ 
ments  to  reduce  error  produced  by  the  end  of  a  measurement 
period  are  discussed  below. 

5.  Multiplexity  F 

The  multiplexity  F  measure  is  concerned  with  forward 
planning  only.  It  considers  the  QIS  value  (see  below)  that 
would  be  calculated  if  only  diagonals  in  the  Time-Event  Matrix 
that  connect  two  points  ahead  in  time  are  considered.  As  a 
result,  the  measure  focuses  on  the  degree  of  strategy  that  is 
presently  applied  and  ignores  strategic  components  that  have 
already  been  completed.  It  should  be  noted,  that  Multiplexity  F 


is  especially  sensitive  to  end  effects  of  data  collection  or 
observation.  The  formula  for  Multiplexitv  F  is  similar  to  the 
formula  for  QIS  (with  earlier  integrations  omitted)  and  may  be 
written  as 

P 

W  ( 1 +n  f  ) 

1 

For  an  explanation  of  terms,  see  Measures  6  and  9. 

A  related  measure  that  is  not  currently  printed  by  the 
simulation  program  but  is  related  to  Multiplexitv  F  is  called 
Multiplicity.  The  Multiplicity  measure  is  also  closely  akin  to 
the  QIS  measure  (Measure  9).  It  does,  however,  not  take  time 
between  decisions  in  an  integration  sequence  (See  measure  6, 
below)  into  account.  While  Multiplicity  is  not  orthogonal  to 
QIS,  it  is  considered  to  be  supplemental.  This  measure  is 
potentially  more  meaningful  than  QIS  in  situations  where  respond 
ing  (including  strategic  integrated  responding)  must  occur  quite 
rapidly  or  where  the  time  delay  between  an  original  and  a  later 
decision  in  strategic  sequences  is  more  a  function  of  task 
demand  than  of  planning  or  task  performance.  The  formula  for 
multiplicity  can  be  derived  directly  from  the  QIS  formula 
(Measure  9)  by  removing  the  term  for  Weight  (Measure  6): 

P 

(1  +  n p  +  nf  ) 

1 


For  an  explanation  of  terms,  see  Measure  9. 


6.  Integration  Time  Weight 

The  Time  Weight  measure  calculates  the  length  of  time  for 
which  plans  and/or  strategies  are  developed. 


Where  the  measure  for  number  of  integrations  was  merely 
concerned  with  the  frequency  of  interconnections  (strategic 
relationships)  among  decisions,  the  time  weight  measure  considers 
the  length  of  time  involved  in  future  planning.  The  measure 
utilizes  the  same  individual  integrations  discussed  for  the 
integration  measure  (diagonals)  but  measures  each  diagonal  on 
the  time  dimension  (in  fixed  units  chosen  by  the  experimenter  or 
observer,  see  above)  and  replaces  the  value  of  1  (for  the 
occurrence  of  the  diagonal)  with  the  value  of  length  of  time 
which  the  diagonal  spans  between  decisions.  Consider,  for 
example,  the  matrix  example  in  Figure  3,  below: 
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The  time  weight  for  the  forward  integrations  (diagonal 
connections)  between  the  initial  decisions  203  and  488  which  are 
connected  to  decision  221  represents  two  time  units  each  (remem¬ 
ber  that  time  units  are  selected  by  the  experimenter/observer  but 
must  be  held  constant  if  numerical  comparisons  among  scores  for 
different  decision  makers  are  to  be  made).  The  connection 
between  decisions  221  and  223  represents  four  time  units.  The 
connection  between  223  and  221  represents  one  time  unit,  and 
finally  the  connection  between  461  and  223  represents  two  time 
units.  The  total  score  for  integration  time  weight  in  this 
matrix  is  then  Ln  ( 2  +  2  +  4  +  1  +  2 )  =  the  natural  log  of  11. 

7.  Backward  Integrations 

The  rationale  for  scoring  backward  integrations  and  a 
formula  for  that  measure  have  already  been  discussed  (cf. 

Measure  4)  and  need  not  be  repeated  at  this  point. 


8.  Unintegrated  Respondent  Decisions  (Retaliatory  Decisions) 
Decisions  or  other  actions  which  occur  in  response  to 
information  may  be  part  of  an  ongoing  strategy  and  may  reflect 
an  understanding  of  the  interrelationship  between  events, 
actions  and  anticipated  outcomes.  Alternatively,  respondent 
actions  may  be  retaliatory  rather  than  strategic,  i.e.,  they  may 
be  one-to-one  responses  to  incoming  information  that  do  not 
consider  the  overall  impact  of  information  on  plans,  strategies 
and  so  forth.  The  latter  actions  are  considered  in  the  measure¬ 
ment  of  unintegrated  respondent  decision  making.  Returning  to 
the  Time-Event  Matrix,  all  respondent  decisions  that  are  not 
interconnec ted  with  other  decisions  by  diagonals  would  be 
included  in  this  measure.  A  formula  for  this  measure  may  be 
written  as 


where 

ru  are  all  decisions  that  are  made  in  response  to  incoming 
information  which  are  not  part  of  an  integrated 
sequence  . 

9.  QIS  (Quality  of  Integrated  Strategies) 

This  measure  provides  an  indication  of  the  complexity  of 
sequential  strategic  efforts.  Similar  data  for  special  applica¬ 
tions  are  represented  by  the  Multiplexity  of  Integration  and  the 
Weighted  QIS  (WQIS)  measures. 


Ln 


W ( 1  +  n  +  n,) 
P  £ 
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Where  W  represents  the  length  of  the  time  dimensions  for  any 
forward  integration  (or  any  integration,  if  distinc¬ 
tions  between  forward  and  backward  integrations  cannot 
be  made).  W  is  the  last  measure  discussed  above 
(integration  time  weight) 

n  p  is  the  number  of  other  forward  integrations  (or  any 

integration,  if  distinctions  between  forward  and 
backward  integrations  cannot  be  made)  connecting  to 
the  decision  point  representing  the  initial  decision 
in  a  diagonal  connection  between  two  decisions  and 
nf  is  the  number  of  forward  integrations  (or  any  integra¬ 
tions,  if  distinctions  between  forward  and  backward 
integrations  cannot  be  made)  connecting  to  the  decision 
point  representing  the  subsequent  decision  in  a 
diagonal  connection  between  two  decisions. 

The  number  of  integrations  np  and  nf  here  include  only  those 
integrations  which  are  directly  connected  to  either  the  initial 
(np)  or  subsequent  (nf)  decision  points. 

The  Q1S  measure  assesses  the  degree  to  which  planning 
(strategic  behavior)  follows  an  overall  pattern  versus  a  number 
of  separate  unrelated  plans.  While  the  score  for  number  of 
integrations  may,  for  example,  be  the  same  in  either  case,  an 
overall  plan  connecting  all  components  of  a  decision-making 
sequence  in  a  combined  strategy  would  result  in  a  higher  QIS 
score;  separate  strategic  plans  would  result  in  lower  QIS  scores. 
QIS  scores  tend  to  distinguish  between  excellent  and  mediocre 


decision-making  quality  where  decision  makers  must  operate  at 
advanced  decision-making  levels.  QIS  are  identical  to  Integra¬ 
tion  Time  Weight  scores  where  all  integrations  remain  independent 
of  each  other,  i.e.,  where  an  overall  strategic  plan  does  not 
exist  . 

Example 

Let  us  again  return  to  Figure  3,  page  31.  A  QIS  value 
would  be  established  for  each  diagonal  in  the  matrix.  Let  us 
initially  take  the  diagonal  which  is  connecting  203  and  221.  We 
already  concluded  that  its  not  yet  transformed  weight  ( W )  score 
is  2.  There  are  no  diagonals  connecting  to  its  beginning  point. 
On  the  other  hand,  there  are  two  diagonals  connecting  directly 
to  its  end  point.  The  score  would  be 
2(1  +0+2)  =6 

The  same  value  of  6  would  also  be  obtained  for  the  488  to  221 
diagonal.  The  221  to  223  diagonal  with  a  W  value  of  4  connects 
to  two  other  diagonals  at  its  beginning  point  and  one  other 
diagonal  at  its  end  point.  Its  score  would  be 
4(1+2+  1)=  16 

In  turn  the  223  to  221  connection  would  be 

1  (  1  +  1  +  0)  =  2 

Finally,  the  461  to  223  diagonal  maintains  its  W  value  since 
there  are  no  diagonals  connected  to  either  the  initial  nor  the 
subsequent  decisions: 

2( 1  +  0  +  0)  =  2 

For  this  matrix  the  total  QIS  score  then  would  be 


Ln  (6  +  6  +  16  +  2  +  2)  =  the  natural  log  of  32 
10.  Weighted  QlS 

Weighted  Quality  of  Integrated  Strategies  (WQIS)  is  an 
extension  of  the  QIS  measure  to  obtain  scores  for  the  sequential 
chain  of  interconnections  among  integrated  decisions  over  long 
periods  of  time  (i.e.,  multiple  long-term  strategic  actions  that 
are  coordinated).  Where  the  QIS  formula  calculated  the  time 
weight  for  an  integration  (diagonal  connection  between  decision 
points  differing  in  time)  and  multiplied  that  weight  value  with 
the  number  of  other  diagonals  connected  directly  to  the  beginning 
point  (initial  decision)  and  the  end  point  (later  integrated 
decisions)  of  an  integration,  the  WQIS  measure  considers  all 
integrations  (diagonals)  which  lead  in  chain  sequences  to  the 
decision  which  begins  the  integration,  as  well  as  all  integra¬ 
tions  (diagonals)  which  follow  the  later  integrated  decision,  as 
long  as  there  is  no  interruption  in  diagonal  (integration) 
links.  Because  of  the  multiplicative  nature  of  this  measure, 
quite  high  scores  can  be  obtained  whenever  additional  links  are 
added  in  any  strategic  chain  of  decisions.  Where  no  more  than 
three  decision  points  (differing  in  time)  are  connected  with 
diagonals  (integrations),  the  WQIS  measure  will  not  differ  from 
the  QIS  measure.  Where  four  decision  points  (three  sequential 
diagonals)  are  involved,  the  measure  will  not  differ  for  the 
middle  integration,  but  will  differ  for  the  outer  two  integration 
diagonals.  With  even  greater  numbers  of  diagonal  connections 
in  chain  sequence,  the  score  for  W Q I  ?  would  considerably  exceed 


the  Q I  S  score.  The  formula  for  WQIS  can  be  written  as 


Ln 


where 


f  ) 


npp  is  the  number  of  forward  integrations  reflected  in  the 
term  np  for  the  QIS  measure  plus  all  other  forward 
integrations  connecting  to  these  integrations,  and  so 
forth,  until  all  integrations  (diagonals  in  the 
matrix)  which  connect  to  each  other  and  can  be  traced 
without  interruption  to  the  beginning  point  of  the 
forward  integration  of  interest  have  been  exhausted, 
nff  is  the  number  of  forward  integrations  reflected  in  the 
term  np  for  the  QIS  measure  plus  all  other  forward 
integrations  connecting  to  these  integrations,  and  so 
forth,  until  all  integrations  (diagonals  in  the 
matrix)  which  connect  to  each  other  and  can  be  traced 
without  interruption  to  the  later  decision  have  been 
exhausted . 

All  other  terms  are  the  same  used  in  previous  formulas 
For  the  example  in  Figure  3  on  page  31  the  WQIS  score  would 
be 

2 ( 1  +  0+3 )  +  2(1 +0+3 )  +  4  ( 1  +  2+1  )  +  1(1  +  3+0 )  +  2(l+0+0)  =  the 
natural  log  of  38 


11.  Average  Response  Speed 

The  measure  reflects  the  typical  time  delay  between  receipt 


of  information  and  decision  responses  to  that  information. 


P 


where 

tr  is  the  elapsed  time  between  information  receipt  and 
any  subsequent  respondent  decision,  and 

r p  is  the  number  of  respondent  decisions  made  in  the  time 
period  between  1  and  p. 

The  response  speed  measure  simply  reflects  the  rapidity 
with  which  decision  maker(s)  respond  to  incoming  information. 

The  time  length  between  each  input  and  the  subsequent  decision 
is  measured;  the  sum  of  those  measures  is  divided  by  the  number 
of  observed  responses  to  information.  For  this  measure  it  is 
worthwhile  to  consider  a  value  for  r  (number  of  respondent 
decisions)  which  is  not  constrained  by  a  time  limitation  between 
information  receipt  and  subsequent  decision. 

If  no  responses  to  information  should  occur  (a  rather 
unlikely  event)  the  score  for  Measure  11  must  be  set  to  a  value 
that  is  higher  than  all  other  scores  in  a  sample. 

12.  Serial  Connections 

The  serial  connection  measure  is  similar  to  the  Number  of 
Integrations  measure  but  counts  interconnections  between  decis¬ 
ions  that  are  placed  into  the  same  decision  category.  For 
example,  if  decision  makers  decide  to  move  troop  unit  A  and  plan 
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to  subsequently  move  troop  unit  B,  a  forward  serial  connection  is 
established:  Both  decisions  fall  into  a  single  decision 

category:  troop  movement.  They  are,  by  themselves,  not  likely 
to  reflect  an  ongoing  strategy  unless  they  are  also  interconnect¬ 
ed  with  other  decisions  located  in  different  categories  (to 
which  they  would  be  connected  with  diagonals  in  the  matrix). 
Serial  connections  without  integrations  often  reflect  a  stagna¬ 
ting  series  of  moves  that  frequently  fail  to  take  the  complex¬ 
ities  of  a  task  or  environment  into  account.  If  associated  with 
strategic  moves  (as  reflected  in  high  scores  on  such  measures  as 
Number  of  Integrations  or  QIS),  they  may  be  part  of  a  general 
(e.g.,  in  the  military,  an  encircling)  strategy. 

Serial  connections  may  be  measured  (as  were  number  of 
integrations)  in  terms  of  forward,  backward  or  general  connec¬ 
tions  between  decisions  of  a  single  category: 


i  s  f 


or 


P 

or  >  ( i s  f  +  isb) 
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where 


isf  are  forward  serial  connections,  and  i  s  b  are  backward 
serial  connections. 

13.  Planned  Integrations 

Not  all  actions,  here  decisions,  which  are  planned  as 


% 
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sequels  to  current  actions  will  actually  be  carried  out.  Time 
demands,  changed  situations,  forgetfulness,  new  strategies  and 
more  may  be  the  cause  of  lacking  follow-up  actions.  Where 


multiple  strategies  toward  goals  were  followed,  planned  actions 
that  are  part  of  strategy  may  be  dropped  as  unnecessary  when 
some  other  parallel  strategy  is  about  to  reach  its  successful 
conclusion.  In  other  cases,  however,  an  incomplete  connection 
between  a  current  action  and  a  planned  future  action  may  indicate 
poor  implementation  of  strategy.  The  Planned  Integrations 
measure  reflects  the  number  of  times  decision  makers  fail  to 
carry  out  subsequent  future  actions  that  had  been  previously 
planned.  The  formula  for  planned  integrations  can  be  written  as 


P 
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where 

ipf  is  a  planned  forward  integration  which  was  not  carried 
out  in  the  future. 

Planned  integrations  that  did  not  come  to  fruition  may  be 
compared  with  the  number  of  integrations  which  were  completed  to 
obtain  an  estimate  of  the  degree  to  which  decision  makers  do 
operationalize  their  plans.  This  score  would  be  reflected  by 
the  formula 


P  P 


Finally,  the  planned  integration  measure  may  be  utilized  to 
estimate  the  assumed  time  value  for  number  of  integrations  where 
that  measure  is  truncated  by  the  end  of  a  measurement  or 


observation  sequence  (e.g.,  at  final  participation  periods  in 
experimental  simulations  or  at  the  retirement  of  an  executive  or 
officer  prior  to  final  completion  of  a  task).  Under  these 
conditions,  it  may  not  have  been  possible  to  complete  all  future 
decisions  which  were  planned  when  a  previous  action  was  initi¬ 
ated.  As  a  result  the  uncorrected  measure  for  number  of 
integrations  would  underestimate  the  strategic  planning  of  the 
decision  maker.  This  correction  may  be  calculated  as: 


PP 
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where  1  through  pp  is  any  prior  time  period  (or  periods)  to  which 
a  time  period  under  analysis  is  to  be  compared. 

The  obtained  value  of  this  correction  is  then  multiplied 
with  the  total  number  of  intended  integrations: 


to  obtain  the  estimated  value  of  corrected  Number  of  Integra¬ 
tions.  Unless  the  corrected  value  is  less  than  the  actually 
obtained  score  for  Number  of  Integrations,  the  Number  of  Integra¬ 
tions  score  may  be  replaced  by  the  corrected  score.  A  measure 


calculating  estimated  values  on  the  basis  of  earlier  simulation 
performance  (Measure  36).  Similar  calculations  may  be  employed 
to  correct  other  measures  which  are  based  on  forward  or  backward 
integrations  . 

14.  General  L’nintegrated  Decisions 

Decisions  which  are  neither  part  of  any  integrated  strategy 
nor  are  made  in  response  to  information  are  defined  as  General 
Unintegrated  Decisions.  In  most  cases,  these  decisions  are  not 
particularly  valuable  unless  they  are  part  of  a  tentative  probe 
for  preliminary  information,  e.g.,  in  the  form  of  "I  wonder  what 
would  happen  if...":  In  a  more  or  less  defined  task  setting 
with  the  potential  for  strategic  goal  directed  actions,  an 
excess  of  General  Unintegrated  Decisions  tends  to  reflect  poor 
performance.  The  formula  for  this  measure  may  be  written  as 
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where  u  is  an  unintegrated  respondent  decision. 

15.  Spread  across  Decision  Categories-* 

This  measure,  as  well  as  the  next  measure  (Average  Spread 
Across  Decision  Categories),  considers  the  degree  to  which 
decision  makers  favor  certain  kinds  of  actions  over  other  kinds. 

^This  measure  has  been  slightly  modified  compared  to 
previous  applications. 
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/  2 ( d C a  ~  deb)  +  (^Cd  -  dce) 

1 

where,  d  is  the  number  of  decisions 

d^a  is  the  number  of  decisions  from  the  category  or 
categories  representing  the  upper  ten  percent  of 
decision  frequency, 

d^b  is  the  number  of  decisions  from  the  category  or 

categories  representing  the  lowest  ten  percent  of 
decision  frequency, 

is  the  number  of  decisions  from  the  category  or 
categories  representing  the  upper  fifty  percent  of 
decision  frequency,  and 

d£e  is  the  number  of  decisions  from  the  category  or 

categories  representing  the  lower  fifty  percent  of 
decision  frequency. 

A  high  value  of  this  measure  suggests  that  the  decision  maker(s), 
while  not  necessarily  totally  ignoring  other  potential  decision 
categories,  nonetheless  concentrate  major  effort  on  a  limited 
number  of  activities.  For  example,  an  executive  who  selects 
most  decisions  because  they  specifically  relate  to  current 
profit  or  a  military  commander  who  lets  the  infantry  do  nearly 
all  of  the  fighting  without  support  by  other  units  would  score 
high  on  this  measure.  A  low  score,  on  the  other  hand,  would 
reflect  a  more  well  rounded  approach  to  more  or  less  complex 
tasks.  The  measure  is  not  meaningful  if  only  one  decision 


category  is  utilized. 

Example 

Let  us  assume  that  decision  maker(s)  made  a  total  of  fifty 
decisions  during  a  given  period  of  time.  These  decisions 
represented  the  following  decision  types: 

Decision  Category  Number  of  Decisions 


1 1 1 

5 

122 

10 

312 

20 

333 

3 

415 

8 

526 

1 

527 

_ 3 

Total  50 


The  upper  and  lower  ten  percent  would  represent  a  value  of 
5  each  (10%  of  fifty)  while  the  upper  and  lower  50%  would 
represent  values  of  25  each. 

Decision  category  312  is  included  in  the  upper  10%,  provid¬ 
ing  a  value  of  20  (the  number  of  decisions  in  that  decision  type 
category)  for  d  c  a  . 

Decision  categories  333,  526  and  527  are  included  in  the 
lower  ten  percent  providing  a  value  of  7  for  d^^. 

Decision  categories  122  and  312  are  included  in  the  upper 
50%,  providing  a  value  of  30  for  d^j . 

Decision  categories  122,  415,  111,  333,  526  and  527  are 


included  in  the  lower  50%  providing  a  value  of  30  for  dQe. 


The  resulting  calculation  for  spread  across  decision 
categories  is  then:  2(20  -  7)  +  (30  -  30)  =  26. 

16.  Average  Spread  Across  Decision  Categories 

The  formula  for  Spread  Across  Decision  Categories  is  to 
some  degree  affected  by  the  number  of  categories  used  (and/or 
available).  To  correct  for  potential  errors,  particularly  when 
many  decision  categories  are  available  and  utilization  differs 
widely  among  individuals  or  groups,  an  additional  measure  may  be 
introduced.  This  measure  divides  the  score  for  spread  across 
categories  by  the  number  of  categories  utilized.  The  average 
spread  measure  is  not  assumed  (or  demonstrated)  to  be  orthogonal 
from  Spread  Across  Decision  Categories.  It  may  be  calculated  as 


P 


1 


For  the  example  of  decision  types  and  decision  frequencies 
on  page  43  the  obtained  numerical  score  would  be  26  (the  value 
obtained  for  the  Spread  Across  Decision  Categories  measure) 
divided  by  the  Number  of  Categories  measure,  here  7  =  3.7143. 

17.  Number  of  Decisions  in  Response 

Where  Measure  2  was  concerned  with  the  number  of  information 
items  which  lead  to  responses,  Measure  17  focuses  only  on  the 
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number  of  responses  themselves  that  were,  at  least  in  part, 
generated  bv  information.  For  example,  Measure  2  counts  each 
time  that  a  previous  message  is  listed  as  one  antecedent.  For 
example,  if  an  executive  makes  three  decisions  and  claims  that 
each  of  those  decisions  was  based  on  three  previous  messages, 
the  resulting  score  for  Measure  2  would  have  been  9.  In  con¬ 
trast,  Measure  17  counts  only  the  decisions  themselves  if  they 
are  claimed  to  be  in  response  to  previous  information,  irrelevant 
of  the  number  of  messages  to  which  those  responses  were  made. 

In  other  words,  the  same  person's  score  on  Measure  17  would 
be  3.  Measure  17  may  be  represented  by  the  formula 
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where  r  equals  the  number  of  decisions  that  listed  previous 
messages  as  the  basis  of  decision  making.  It  differs  from 
Measure  2  in  the  term  r  which,  in  that  case,  was  r^. 

18.  Most  Recent  Response  Speed 

Measure  11  had  assessed  the  mean  response  speed  during  any 
one  particular  period  of  interest.  The  score  would  be  a  reflec¬ 
tion  of  both  immediate  and  delayed  responding  to  information. 
However,  some  situations  do  not  lend  themselves  to  respondent 
action  until  some  critical  event  has  occurred.  Once  that  event 


is  observed,  an  immediate  response  might  only  then  become 
optimal.  Measure  11  might,  in  some  situations,  confound  the 
assessment  of  the  latter  response  with  elapsed  time  between 


earlier,  partially  relevant  information  and  the  arrival  of  later 
critical  information.  That  time  span  is,  however,  under  control 
of  the  simulation  system  and  is  not  affected  by  any  actions  of 
participants.  Consequently,  it  seemed  useful  to  develop  a 
measure  that  would  assess  the  rapacity  of  response  to  the  last 
relevant  message  received.  Measure  18  was  designed  for  that 
purpose : 
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where 

trr  is  the  time  elapsed  between  receipt  of  the  most 

recent  relevant  information  and  the  subsequent  respond¬ 
ent  decision  and 

r p  is  (as  it  was  in  Measure  11  )  the  number  of  respondent 

decisions  during  the  time  between  1  and  p. 

If  no  respondent  decisions  occur,  the  score  for  Measure  18 
should  be  set  to  a  value  higher  than  all  other  scores  obtained 
from  the  sample. 

19.  Total  Integrative  Activity  (TIA) 

Total  Integrative  Activity  is  a  measure  of  strategic 
effort.  It  includes  a  count  of  attempted  integrations  that  did 
not  reach  fruition  and  completed  integrations.  The  latter 
category  includes  forward  integrations  that  are  based  on  planning 


as  well  as  backward  integrations  that  derive  from  retrospective 
pragmatic  insight.  TIA  scores  are  calculated  as  the  simple  sum 
of  Measure  4  (Forward  Integrations),  Measure  7  (Backward  Integra¬ 
tions),  Measure  12  (Serial  Connections)  and  Meat  ;re  13  (Planned 
Integrations).  Persons  scoring  high  on  Measure  4  are  most 
likely  to  score  high  on  this  measure.  However,  many  persons 
scoring  high  on  TIA  fail  to  score  high  on  Measures  4  or  Measure 
12. 

20.  Total  Forward  Integrative  Activity  (TFIA) 

Total  Forward  Integrative  Activity  is  similar  to  Measure 
19,  but  considers  only  planned  integrative  actions.  The  measure 
is  calculated  as  a  simple  sum  of  Measures  4  and  13. 

21.  Integrations  Across  Categories 

Earlier,  the  method  of  assigning  three  digit  numbers  to 
decision  categories  was  discussed.  The  reader  will  remember 
that  the  first  digit  represents  a  general  characteristic  (e.g., 
economic,  military,  etc.).  Measure  21  was  designed  to  assess 
the  degree  to  which  integrations  (strategic  actions),  as  discus¬ 
sed  in  Measure  4,  extend  beyond  the  first  digit  level  of  decision 
categories.  For  example,  if  one  economic  decision  was 
strategically  integrated  with  another  economic  decision,  the 
score  for  that  integration  on  Measure  21  would  be  zero.  However, 
a  score  of  one  would  be  generated  if  the  economic  decision  was 
integrated  with  a  military  decision,  and  so  forth.  In  other 


words,  Measure  21  estimates  the  degree  to  which  strategic 
(integrative)  planning  is  more  generalized  or  more  limited  in  its 
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range.  The  formula  for  Measure  21  has  some  similarity  to  the 
formula  for  Measure  4: 
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with  the  term  if  of  Measure  4  replaced  by  the  term  ifc 

ifc  is  a  forward  integration  (relationship,  i.e.,  connec¬ 
tion  between  decision  making  points  in  the  time-event 
matrix  with  arrowheads  pointing  forward)  i_f  this 
connection  interrelates  decisions  from  categories  with 
a  different  first  numerical  digit. 

22.  Integrations  Within  Categories 

This  Measure  calculates  the  frequency  of  integrative 
interrelationships  that  interconnect  decision  points  that  are 
located  within  the  same  first  digit  decision  category.  While  a 
high  score  in  this  category  does  not  suggest  a  lesser  capacity 
for  integrative  strategic  thought  than  would  be  implied  by  a 
high  score  in  Measure  21,  a  disproportionately  high  score  in 
Measure  22  compared  to  Measure  21,  may  represent  a  lower  level 
of  strategic  thinking.  Where  such  a  lower  level  is  either 
inappropriate  or  less  effective,  considering  the  task  at  hand  a 
diminished  proportion  of  scores  in  Measure  21  over  Measure  22 
would  imply  lower  levels  of  performance.  Measure  22  can  be 


calculated  as 


where 


are  forward  integrations  that  interconnect  decisions 
located  within  the  same  first  digit  decision  category. 


ifw 


Alternately,  Measure  22  may  be  calculated  by  subtracting 
Measure  21  from  Measure  4. 

P  P 


1  1 


23.  Proportion  of  Category  Integrations 

As  already  suggested,  the  proportion  of  Measure  21  divided 
by  Measure  22  may,  if  tasks  are  fluid,  multifaceted  and  complex, 
reflect  the  overall  quality  of  integrative  activity.  Measure  23 
assesses  that  quality.  It  is  calculated  as 

1  +  Measure  21 

1  +  Measure  22 

24.  Shift  1 

This  measure  assesses  the  capacity  of  decision  maker(s)  to 
shift  their  emphasis  from  an  integrative  strategic  mode  to  a 
more  respondent  mode  when  task  demands  require.  For  the 
simulations  employed  by  the  present  research  team,  an  emergency 
is  introduced  at  the  beginning  of  the  fourth  playing  period  in 
the  simulation  (approximately  after  four  hours  of  participation) 
That  emergency  is  best  handled  by  immediate  respondent  actions 


and  without  further  strategic  planning  (for  the  time  being). 

The  calculation  procedure  for  the  Shift  1  measure  would  differ. 


depending  on  scenario  characteristics.  In  our  simulations,  no 
emergencies  occur  throughout  Periods  1,  2,  3  and  6.  Periods  4 
and  5  present  a  disaster  situation  which  requires  respondent 
activity.  In  the  case  of  our  simulation  design,  the  formula  for 
Shift  1  would  be : 

X  Measure  4  (for  Periods  1,2,3)  X  Measure  4  (Periods  4,5 

divided  bv 


A  high  score  would  indicate  the  presence  of  strategic  integrative 
activity  before  the  disaster  event  and  a  shift  to  little  integra¬ 
tive  activity  throughout  the  disaster. 

A  number  of  potential  problems  may  emerge  for  this  measure 
(and  some  subsequent  measures)  that  must  be  controlled  by  the 
introduction  of  default  values.  For  example,  the  complete 
absence  of  integrative  actions  during  Periods  4  and  5  would 
result  in  a  division  of  0  +  0  by  2,  resulting  in  zero.  The 
subsequent  division  of  the  first  part  of  the  formula  by  zero 
would  not  be  meaningful.  Similarly,  some  (especially  the 
second)  part  of  this  formula  may  generate  a  quotient  between 
zero  and  1,  generating  a  meaningless  increase  in  the  score  for 
this  measure.  To  eliminate  these  problems,  a  minimum  default 
value  of  1.0  is  set  for  both  the  first  and  the  second  part  of 
the  formula. 

25.  Shift  2 

This  measure  corrects  Measure  24  for  the  actual  amount  of 


activity  a  person  has  expended  on  respondent  actions  throughout 
the  task.  Calculation  of  the  Shift  2  score  is  based  on  the 
proportion  of  integrative  activity  and  respondent  activity  as 
that  proportion  shifts  from  time  periods  where  strategic  planning 
should  be  a  priority  to  time  periods  where  respondent  activity 
should  be  a  priority.  For  the  simulation  procedures  employed  in 
this  research  (with  planning  appropriate  in  periods  1  through  3 
and  respondent  actions  most  appropriate  in  periods  4  and  5)  the 
formula  should  be 

X  Measure  4  X  Measure  4 

_ (for  periods  1,2,3)  divided  by _ (periods  4,5 

X  Measure  17  X  Measure  17 

Similarly  to  other  measures  which  are  based  on  changes  in 
situational  demands,  the  formula  for  Shift  2  must  be  modified 
with  changes  in  the  introduction  and  duration  of  emergency 
conditions.  Default  values  must  again  be  introduced  if  Measure  4 
or  Measure  17  values  are  equal  to  zero  resulting  in  division 
by  zero  or  division  by  a  fraction. 

26.  Recovery  1 

If  stress  or  emergency  situations  require  a  priority  shift 
from  strategic  to  respondent  activity,  planning  and  strategy 
must  often  re-emerge  once  immediate  problems  have  been  resolved. 
However,  some  persons  may  be  overwhelmed  by  the  emergency  stress 
experience  and  may  fail  to  recover,  i.e.,  may  fail  to  reinstitute 
strategic  and  planning  activities  when  appropriate.  Those 
persons  would  not  perform  adequately  in  task  settings  that 
contain  intermittent  stress  periods.  To  assure  that  assessment 


of  performance  does  not  miss  the  potential  detrimental  impact  of 
stressor  experience,  two  recovery  measures  were  developed. 
Recovery  1  compares  the  number  of  integrations  after  an  emergency 
has  been  resolved  (period  6  in  the  present  simulations)  with  the 
number  of  integrations  which  were  evident  prior  to  the  stressor 
experience  (before  the  emergency  occurred).  For  the  present 
simulations,  the  score  is  calculated  as  follows: 

Measure  4  (Period  6) 

X  Measure  4  (Periods  1,  2  and  3) 

Where  the  Mean  value  of  Measure  4  for  periods  1  through  3  equals 
zero,  the  score  for  this  Measure  is  equal  to  the  value  of 
Measure  4  in  Period  6. 

27.  Number  of  Information  Search  Decisions 

This  measure  simply  counts  the  number  of  information  search 
activities  in  which  decision  makers  engage  during  a  time  period 
of  interest : 
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Where 

s  represents  a  decision  with  the  purpose  of  obtaining 
additional  information. 

28.  Number  of  Integrated  Information  Search  Decisions 

This  measure  is  concerned  with  the  purpose  of  information 
search.  Where  information  search  was  initiated  for  strategic 
reasons,  the  search  action  will  typically  precede  subsequent 


related  actions,  i.e.,  it  will  be  integrated  with  those  later 
actions.  In  the  time-event  matrix,  a  forward  arrow  will  appear 
between  the  search  decision  and  subsequent  related  decision(s). 
The  formula  may  be  written  as 
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Where 

ifis  are  the  number  of  forward  integrations  that  originate 
in  information  search  decisions. 

29.  Disaster  Response  Speed 

Measures  29  through  35  focus  on  the  introduction  of  an 
emergency  that  is  an  intricate  part  of  our  own  simulation 
procedure.  Where  emergencies  are  not  introduced,  Measures  29 
through  35  are  meaningless.  The  advent  of  an  emergency  in  our 
simulations  is  coded  by  a  message  with  the  number  M300.  The 
specified  time  location  of  M300  within  the  simulation  allows  the 
calculation  of  values  for  emergency  responses.  The  first  of 
these,  Disaster  Response  Speed,  measures  the  elapsed  time 
between  the  advent  of  the  emergency  message  M300  and  the  first 
respondent  decision  to  that  message.  The  formula  may  be  written 
as  follows: 

tdl  =  tri  -  tM3Q0 

Where 

tdl  is  the  obtained  disaster  response  speed 

tri  is  the  time  of  the  first  respondent  decision  relevant 
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to  the  emergency  message,  and 

tyj3Q0  is  the  time  of  arrival  of  Message  300  (the  emergency 
message  ) 

If  no  response  to  the  emergency  message  M300  is  made,  the 
score  for  Measure  29  must  be  set  to  a  value  that  is  higher  than 
all  other  scores  in  the  sample. 

30.  Average  Disaster  Response  Speed 

Most  decision  makers  respond  to  the  emergency  message  with 
a  number  of  actions.  The  present  measure  calculates  the  mean 
time  distance  between  the  arrival  of  the  emergency  message  and 
the  various  responses  to  that  message: 

t Xd  =  *(cr  -  tM300 ) 

Where 

txd  is  the  obtained  Mean  Disaster  Response  Speed 
tr  is  the  time  of  any  decision  which  responds  to  Message 
300,  and 

tM300  i  s  the  time  of  arrival  of  the  emergency  message 
( M300 ) 

If  no  response  to  the  emergency  message  M300  is  made,  the 
score  for  Measure  29  must  be  set  to  a  value  that  is  higher  than 
all  other  scores  in  the  sample. 

31.  Integrated  Disaster  Responses 

This  measure  considers  the  degree  to  which  the  potential  of 
an  emergency  was  included  in  previous  strategic  planning.  If 
this  measure  is  to  be  meaningful,  the  simulation  must  be  designed 
so  that  emergency  (or  equivalent)  problems  are  viewed  as  possible 


by  participants  and  can  be,  to  some  extent,  anticipated  in  their 
potential  characteristics.  This  is  the  case  for  the  simulations 
we  have  designed.  The  measurement  for  integrated  disaster 
response  counts  the  number  of  forward  diagonals  that  end  (termin¬ 
ate)  in  any  decisions  that  list  the  disaster  message  (M300)  as  a 
preceding  relevant  message: 

^  1 f  r M300 

Where 

ifrM300  are  a 1 1  integrations  that  terminate  in  respondent 
decisions  based  (entirely  or  in  part)  on  Message  300. 
32.  Applied  Disaster  Strategy 

While  strategic  actions  in  response  to  emergency  situations 
are  not  likely  and  often  are  not  appropriate,  some  integrated 
strategies  might  reasonably  originate  from  responses  to  an 
emergency  event.  The  Applied  Disaster  Strategy  measure  was 
designed  to  assess  the  number  of  strategic  integrations  (forward 
diagonals  in  the  matrix)  that  originate  from  respondent  decisions 
which  list  Message  300  as  a  relevant  preceding  message.  The 
formula  may  be  written  as: 

ifriM300 

Where 

ifriM300  are  integrations  that  originate  from  respondent 

decisions  that  list  the  disaster  message  M300  as  a 


preceding  message. 


33.  Number  of  Disaster  Decisions 


We  are  here  concerned  with  the  number  of  respondent  actions 
to  the  advent  of  an  emergency.  The  measure  simply  counts  the 
number  of  respondent  decisions  (whether  or  not  they  are  inte¬ 
grated)  that  list  M300  as  a  relevant  preceding  message.  The 
formula  may  be  written  as  follows: 


dM300 

Where 

dM300  i-s  any  decision  made  in  response  to  Message  300. 

34.  Disaster  Backward  Integrations 

Actions  taken  at  an  earlier  time,  while  not  previously 
planned  to  deal  with  potential  future  emergencies,  might  nonethe¬ 
less  be  useful  once  an  emergency  arises.  Backward  integrations 
(i.e.,  pragmatic  post-hoc  application  of  strategy)  that  are 
applied  to  actions  which  respond  to  emergencies  are  counted  in 
this  measure.  The  score  is  obtained  with  the  following  formula: 


Where 


bM300 


ibM300  are  backward  integrations  connected  to  respondent 

decisions  that  list  Message  300  as  a  preceding  message. 

35.  Disaster  TIA 

We  have  already  encountered  TIA  in  Measure  19.  In  this 
case  we  are  only  concerned  with  Total  Integrative  Activity  as  it 
applies  to  the  emergency  situation.  Disaster  TIA  calculates  a 
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II A  value  only  for  those  respondent  decisions  that  list  M300  as 
a  preceding  message.  The  formula  for  Disaster  II A  is  calculated 
as 

X —  .  \ -  . 

1  f  r  iM300  +  i  bM300  +  /  ipfM300  +  ■'  isfM300 

Z _  zl -  z- -  zi - 

where  the  first  two  terms  are  applied  as  previously  discussed  and 
*pfM300  are  planned  forward  integrations  that  were  not 

realized  and  were  planned  to  originate  in  a  respondent 
decision  to  Message  300. 

isfM300  are  serial  connections  within  the  same  three  digit 
decision  category  that  were  realized  and  originated  in 
respondent  decisions  to  Message  300. 

36.  Adjusted  Number  of  Forward  Integrations 

This  measure  was  designed  to  correct  for  a  potential  end 
effect  that  might  diminish  the  number  of  integrations  obtained 
in  later  periods  of  any  controlled  decision  making  task  (e.g.,  a 
simulation)  or  that  may  be  due  to  the  termination  of  observations 
in  an  executive  setting.  Measure  36  calculates  the  proportion 
of  realized  and  planned  integration  in  earlier  periods  of  time 
and  applies  that  proportion  to  generate  expected  values  during 
later  time  periods.  This  formula  must,  of  course,  be  adapted  to 
the  length  of  any  simulation  or  period  of  observation.  For 
present  purposes,  the  first  two  periods  of  participation  in  our 
simulations  are  viewed  as  a  performance  baseline.  Corrections 
are  applied  to  all  subsequent  periods.  To  obtain  the  value  for 
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the  baseline  proportion,  the  value  Y  is  calculated  as 

Y  =  X  of  _ Measure  4 _  (for  periods  1  and  2) 

Measure  20 

To  obtain  a  correction  coefficient  Z,  V  is  multiplied  by  Measure 
20  for  each  of  the  subsequent  periods.  The  value  of  Measure  36 
for  each  time  segment  (simulation  period)  is  the  larger  of 
either  Measure  4  or  the  calculated  value  of  Z.  Measure  36 
obtains  this  score  for  periods  3-6  of  the  simulation.  As  stated 
earlier,  scores  for  periods  1  and  2  are  set  to  be  identical  to 
scores  obtained  by  Measure  4. 

Occasionally ,  the  formula  may  call  for  decision  by  zero. 
Where  that  is  the  case  the  value  divided  by  zero  is  set  to  zero. 
37.  Recovery  2 

This  measure  is  similar  to  Recovery  1  (Measure  26)  but 
employs  Measure  36  rather  than  Measure  4  as  the  numerator.  In 
other  words,  it  adjusts  the  obtained  score  via  the  adjusted 
(calculated)  number  of  forward  integrations,  if  they  exceed  the 
actual  number  of  integrations  in  later  periods  of  observation  or 
task  performance.  For  the  simulations  employed  in  our  research, 
the  formula  for  Recovery  2  is 

Measure  36  (period  6) 

X  Measure  4  (for  periods  1,  2  and  3) 

If  the  numerator  in  the  formula  falls  below  a  value  of  1.0,  the 
latter  will  be  substituted  to  avoid  an  unwarranted  increase  in 
the  obtained  score  for  the  Recovery  2.  In  other  words,  if  X 
Measure  4  for  periods  1,  2  and  3  =  3.0,  the  value  of  Measure  37 


will  be  equal  to  Measure  36. 

Additional  Measurement 

(1)  Frequency  Proportions 

To  some  extent,  performance  characteristics  of  a  person  or 
a  decision  making  group  may  be  limited  by  the  overall  decision 
or  action  output  in  a  task.  These  limitations  could  be  due 
either  to  lack  of  attention,  to  a  lack  of  opportunity  to  respond 
or  to  task  characteristics  that  may  limit  actions.  Consequently, 
the  proportion  of  performance  score  values  as  part  of  total 
relevant  activity  may,  in  some  cases,  be  a  preferred  measure  of 
performance.  For  that  purpose,  the  quotient  of  relevant  measures 
divided  by  the  number  of  actions  taken  (e.g.,  decisions  made) 
could  represent  an  important  estimate  of  actual  performance. 

That  quotient  is  calculated  in  our  computerized  measurement 
system  for  Measures  2,  4,  5,  7,  8,  19,  20,  21,  22,  27,  28  and  36. 

(2)  Time  Periods  and  Overall  Performance 

Performance  values  on  most  measures  is  calculated  by  the 
IBM  AT  based  software  program  for  six  separate  time  periods  in 
the  simulation.  In  addition,  it  is  summarized  for  the  entire 
simulation.  All  measures  are  included  in  the  score  summary. 
Measure  values  that  are  relevant  to  only  one  event  in  the 
simulation  (e.g.,  the  occurrence  of  an  emergency)  are,  of 
course,  calculated  only  once  and  are  provided  only  as  overall 
values  in  the  final  score  summary.  Scores  that  are  continuously 
relevant  are  provided  for  each  period  and  in  the  final  summary 
table.  An  example  of  such  a  printout  is  provided  in  the  Appendix. 


Since  the  underlying  simulation  software  was  designed  to  be 
flexible,  other  scenarios  may  be  designed  to  utilize  that  program 
at  will.  Such  scenarios,  while  they  might  require  modifications 
of  certain  measures  (as  indicated)  would  also  permit  the 
development  of  additional  measures  of  interest  that  might  be 
based  on  data  that  are  automatically  collected  by  the  simulation 
system . 

The  simulation  technique  permits  the  introduction  of 
measures  that  would  assess  the  cor r ectness  and  timeliness  of  the 
content  of  decisions  and  other  actions  by  participants.  Those 
measures  would,  of  course,  have  to  be  specifically  designed  for 
each  particular  scenario  and  would  likely  require  additional 
programming  to  permit  computer  based  data  collection. 


APPENDIX 

The  following  example  of  a  computer  printout  of  Measures 
from  a  simulation  (taken  from  the  reliability  sample)  contains 
information  on  six  playing  periods  and  a  final  Measure  summary 


PERIOD  1 


9009 D I S  EMCTRL 


1 - MEASURE5*  18  (4  OF  DECISIONS) 

2- MEASURE=  19  105  7.  (#  OF  RESPONDENT  DEC.  > 

3- MEASURE®  15  (4  OF  DEC.  CATEGORIES) 

4- MEASURE®  12  66  7.  (4  OF  FWD  INTEGRATIONS) 

5- MEASURE®  28  155  7.  (MULTIPLEX  I TY  F) 

6- MEASURE®  451  MINUTES  (WEIGHT) 

7- MEASURE®  6  33  7.  (4  OF  BKD  INTEG) 

8- MEASURE®  9  50  7.  (4  OF  UN  INTEG.  RES.  DEC.  ) 

9- MEASURE®  1294  (QIS) 

10- MEASURE®  1937  (WEIGHTED  QIS) 

11 - MEASURE®  4.968422  (AVE. RESPONSE  SPEED) 

12- MEASURE®  2  (SERIAL  CONNECTIONS) 

13- MEASURE®  25  (PLANNED  INTEGRATIONS) 

14- MEASURE®  3  (GENERAL  UNINTEGRATED  DEC.) 

17- MEASURE®  15  83  7.  (4  OF  DECISIONS  IN  RESPONSE) 

18- MEASURE®  3.946667  (MOST  RECENT  RESPONSE  SPEED) 

19- MEASURE®  45  250  7.  (TOTAL  INTEGRATIVE  ACTIVITY  =  TIA> 

20- MEASURE®  37  205  7.  (TOTAL  FORWARD  INTEGRATIVE  ACT  I VI  TY=TF  I  A) 

21 - MEASURE®  10  55  7.  (INTEGRATIONS  ACROSS  CATEGORIES) 

22- MEASURE®  2  117.  (INTEGRATIONS  WITHIN  CATEGORIES) 

23- MEASURE®  3.666667  (PROP. CAT. INTEG. ) 

27- MEASURE®  14  77  7.  (4  INFO  SEARCH  DEC.) 

28- MEASURE®  8  44  7.  (4  INFO  SEARCH  DEC.  INT.  ) 

36-MEASURE®  12  67  7.  (ADJUSTED  4  OF  FWD.  INTEG.  ) 


====================  PERIOD  2  9009DIS  EMCTRL  ============== 

1 - MEASURE*  14  <#  OF  DECISIONS) 

2- MEASURE*  14  100  7.  (#  OF  RESPONDENT  DEC.) 

3- MEASURE*  9  <#  OF  DEC.  CATEGORIES) 

4- MEASURE*  22  157  7.  (#  OF  FWD  INTEGRATIONS) 

5- MEASURE*  145  1035  7.  (MULT  IF'LEX  I TY  F) 

6- MEASURE*  184  MINUTES  (WEIGHT) 

7- MEASURE*  4  28  7.  (#  OF  BKD  INTEG) 

8- MEASURE*  6  42  7.  (#  OF  UNINTEG.  RES.  DEC.  ) 

9- MEASURE*  1598  (QIS) 

10- MEASURE*  5238  (WEIGHTED  QIS) 

11- MEASURE*  10.19286  (AVE. RESPONSE  SPEED) 

12- MEASURE*  7  (SERIAL  CONNECTIONS) 

13- MEASURE*  26  (PLANNED  INTEGRATIONS) 

14- MEASURE*  1  (GENERAL  UN INTEGRATED  DEC.) 

17- MEASURE*  13  92  7.  <#  OF  DECISIONS  IN  RESPONSE) 

18- MEASURE*  8.976923  (MOST  RECENT  RESPONSE  SPEED) 

19- MEASURE*  59  421  7.  (TOTAL  INTEGRATIVE  ACTIVITY=TI A) 

20- MEASURE*  48  342  V.  (TOTAL  FORWARD  INTEGRATIVE  ACTIVITY=TFIA) 

21 - MEASURE*  19  135  7.  (INTEGRATIONS  ACROSS  CATEGORIES) 

22- MEASURE*  3  21  7.  (INTEGRATIONS  WITHIN  CATEGORIES) 

23- MEASURE*  5  (PROP. CAT. INTEG. ) 

27- MEASURE*  1  7  7.  <#  INFO  SEARCH  DEC.) 

28- MEASURE*  0  0  7.  (#  INFO  SEARCH  DEC.  I  NT.  ) 

36-MEASURE*  22  157  7.  (ADJUSTED  #  OF  FWD.  INTEG.  ) 
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====================  PERIOD  3  9009DIS  EMCTRL  ============== 

1-  MEASURE =  13  (#  OF  DECISIONS) 

2- MEASURE=  22  169  */.  (#  OF  RESPONDENT  DEC.  ) 

3- MEASURE*  10  (#  OF  DEC.  CATEGORIES) 

4- MEASURE=  25  192  7.  (#  OF  FWD  INTEGRATIONS) 

5- MEASURE*  220  1692  7.  (MULTIPLEX I TY  F) 

6- MEASURE*  303  MINUTES  (WEIGHT) 

7- MEASURE*  12  92  ’/.  (#  OF  BKD  INTEG) 

8- MEASURE*  10  76  V.  <#  OF  UNINTEG.  RES.  DEC.  ) 

9- MEASURE*  4243  (01 S) 

10- MEASURE*  12838  (WEIGHTED  QIS) 

1 1- MEASURE*  7.045455  (AVE. RESPONSE  SPEED) 

-MEASURE*  3  (SERIAL  CONNECTIONS) 

-MEASURE*  28  (FLANNED  INTEGRATIONS) 

-MEASURE*  2  (GENERAL  UN INTEGRATED  DEC.) 

-MEASURE*  13  100  7.  (#  OF  DECISIONS  IN  RESPONSE) 

-MEASURE*  4.400001  (MOST  RECENT  RESPONSE  SPEED) 

-MEASURE*  68  523  V.  (TOTAL  INTEGRATIVE  ACTIVITY=TIA) 

-MEASURE*  53  407  V.  (TOTAL  FORWARD  INTEGRATIVE  ACTI VITY=TFIA> 

21- MEASURE*  17  130  7.  (INTEGRATIONS  ACROSS  CATEGORIES) 

22- MEASURE*  8  61  7.  (INTEGRATIONS  WITHIN  CATEGORIES) 

23- MEASURE*  2  (PROP. CAT. INTEG. ) 

27- MEASURE*  2  15  7.  (#  INFO  SEARCH  DEC.) 

28- MEASURE*  0  0  7.  (#  INFO  SEARCH  DEC.  INT.  ) 

36-MEASURE*  25  192  7.  (ADJUSTED  #  OF  FWD.  INTEG.  ) 


====================  PERIOD  4  9009DIS  EMCTRL  ==================== 


1-  MEASURE* 

2- MEASURE* 

3- MEASURE* 

4- MEASURE* 

5- MEASURE* 

6- MEASURE* 

7- MEASURE* 

8- MEASURE* 

9- MEASURE* 

10- MEASURE 

1 1 - MEASURE 

12- MEASURE 

13- MEASURE 

1 4- MEASURE 

17- MEASURE 

18- MEASURE 

1 9- MEASURE 

20- MEASURE 

21 - MEASURE 

22- MEASURE 

23- MEASURE 

27- MEASURE 

28- MEASURE 
36-MEASURE 


23  (#  OF  DECISIONS) 

43  186  7.  (#  OF  RESPONDENT  DEC.) 

18  (#  OF  DEC.  CATEGORIES) 

25  108  7.  <#  OF  FWD  INTEGRATIONS) 

129  560  7.  (MULTIPLEX  I  TV  F) 

79  MINUTES  (WEIGHT) 

42  182  7.  (#  OF  BKD  INTEG) 

28  121  7.  (#  OF  UNINTEG.RES.  DEC.  ) 

734  (QIS) 

=  2162  (WEIGHTED  QIS) 

=  16.75814  (AVE. RESPONSE  SFEED) 

=  1  (SERIAL  CONNECTIONS) 

=  19  (PLANNED  INTEGRATIONS) 

=  2  (GENERAL  UN INTEGRATED  DEC. > 

=  23  100  7.  (#  OF  DECISIONS  IN  RESPONSE) 

=  7..30434  (MOST  RECENT  RESPONSE  SPEED) 

=  87  378  7.  (TOTAL  INTEGRATIVE  ACT  I  VITY=TIA> 

=  44  191  7.  (TOTAL  FORWARD  INTEGRATIVE  ACTIVITY=TFIA) 

=  17  73  7.  (INTEGRATIONS  ACROSS  CATEGORIES) 

=  8  34  7.  (INTEGRATIONS  WITHIN  CATEGORIES) 

=  2  (PROP. CAT. INTEG. ) 

=  0  0  7.  (#  INFO  SEARCH  DEC.  ) 

=  0  0  7.  (#  INFO  SEARCH  DEC.  I  NT.) 

=  25  109  7.  (ADJUSTED  #  OF  FWD.  INTEG.  ) 


PERIOD 


9009DIS  EMCTRL 


1 - MEASURE*  15  (#  OF  DECISIONS) 

2- MEASURE*  7  46  7.  (#  OF  RESPONDENT  DEC.) 

3- MEASURE*  12  (#  OF  DEC.  CATEGORIES) 

4- MEASURE*  2  13  7.  (#  OF  FWD  INTEGRATIONS) 

5- MEASURE*  2  13  7.  (MULTIPLEXITY  F) 

6- MEASURE*  2  MINUTES  (WEIGHT) 

7- MEASURE*  4  26  7.  <#  OF  BKD  INTEG) 

B-MEASURE*  6  40  7.  (#  OF  UN  INTEG.  RES.  DEC.  ) 

9- MEASURE*  2  (QIS) 

10- MEASURE*  2  (WEIGHTED  01 S) 

11 - MEASURE*  13.21428  ( A VE. RESPONSE  SPEED) 

12- MEASURE*  4  (SERIAL  CONNECTIONS) 

13- MEASURE*  2  (PLANNED  INTEGRATIONS) 

14- MEASURE*  9  (GENERAL  UN INTEGRATED  DEC. ) 

17- MEASURE*  7  46  7.  (#  OF  DECISIONS  IN  RESPONSE) 

18- MEASURE*  13.21428  (MOST  RECENT  RESPONSE  SPEED) 

19- MEASURE*  12  80  7.  (TOTAL  INTEGRATIVE  ACTIVITY*TIA> 

20- MEASURE*  4  26  7.  (TOTAL  FORWARD  INTEGRATIVE  ACTIVITY  =  TFI A) 

21 - MEASURE*  2  13  7.  (INTEGRATIONS  ACROSS  CATEGORIES) 

22- MEASURE*  0  0  7.  (INTEGRATIONS  WITHIN  CATEGORIES) 

23- MEASURE*  3  (PROP. CAT. INTEG. ) 

27- MEASURE*  4  26  7.  (#  INFO  SEARCH  DEC.  ) 

28- MEASURE*  0  0  7.  (#  INFO  SEARCH  DEC.INT.) 

36-MEASURE*  2  13  X  (ADJUSTED  #  OF  FWD. INTEG. > 


PERIOD  6  9009DIS  EliCTRL 


1 - MEASURE*  16  (4  OF  DECISIONS) 

2- MEASURE*  9  56  7.  (4  OF  RESPONDENT  DEC.) 

3- MEASURE*  12  (4  OF  DEC.  CATEGORIES) 

4- MEASURE*  5  31  7.  (4  OF  FWD  INTEGRATIONS) 

5- MEASURE*  9  56  */.  (MULTIPLEX I TY  F) 

6- MEASURE*  7  MINUTES  (WEIGHT) 

7- MEASURE*  7  43  7.  (4  OF  BKD  INTEG) 

8- MEASURE*  5  31  7.  (4  OF  UNINTEG.  RES.  DEC.  ) 

9- MEASURE*  17  (QIS) 

10- MEASURE*  17  (WEIGHTED  QIS) 

11- MEASURE*  28.76667  (AVE. RESPONSE  SPEED) 

12- MEASURE*  4  (SERIAL  CONNECTIONS) 

13- MEASURE*  2  (PLANNED  INTEGRATIONS) 

14- MEASURE*  5  (GENERAL  UN INTEGRATED  DEC. ) 

17- MEASURE*  9  56  7.  (4  OF  DECISIONS  IN  RESPONSE) 

18- MEASURE*  28.76667  (MOST  RECENT  RESPONSE  SPEED) 

19- MEASURE*  18  112  7.  (TOTAL  INTEGRATIVE  ACTIVITY*TI A) 

20- MEASURE*  7  43  7.  (TOTAL  FORWARD  INTEGRATIVE  ACT I V I T  Y=TF\I  A ) 

21- MEASURE*  3  18  7.  (INTEGRATIONS  ACROSS  CATEGORIES) 

22- MEASURE*  2  12  7.  (INTEGRATIONS  WITHIN  CATEGORIES) 

23- MEASURE*  1.333333  (PROP. CAT. INTEG. ) 

27- MEASURE*  7  43  7.  (4  INFO  SEARCH  DEC.) 

28- MEASURE*  1  6  7.  (4  INFO  SEARCH  DEC.  INT.  > 

36-MEASURE*  5  31  7.  (ADJUSTED  4  OF  FWD.  INTEG.  ) 


OVERALL  MEASURES  9009DIS  EMCTRL 


1 - MEASURE*  99  (#  OF  DECISIONS) 

2-  MEASURE =  114  110  %  (tt  OF  RESPONDENT  DEC.) 

3- MEASURE*  13  (#  OF  DEC.  CATEGORIES) 

4- MEASURE*  91  95  7.  (#  OF  FWD  INTEGRATIONS) 

5- MEASURE=  533  585  7.  (MULTIPLEX ITY  F) 

6- MEASURE*  1026  MINUTES  (WEIGHT) 

7- MEASURE=  75  67  7.  (#  OF  BKD  INTEG) 

Q-MEASURE*  64  60  7.  (#  OF  UNINTEG.  RES.  DEC.  ) 

9-  MEASURE=  7888  (QIS) 

1 0- ME ASURE=  22244  (WEIGHTED  QIS) 

1 1 - MEASURE3  13.49097  ( A VE. RESPONSE  SPEED) 

12- MEASURE*  21  (SERIAL  CONNECTIONS) 

13- MEASURE*  102  (PLANNED  INTEGRATIONS) 

1 4-  MEASURE3  22  (GENERAL  UNINTEGRATED  DEC.) 

1 5- MEASURE*  1  (SPREAD  ACROSS  DEC. CAT.) 

16- MEASURE=  1.785714E-02  (AVERAGE  SPREAD) 

17- MEASURE*  80  80  7.  (#  OF  DECISIONS  IN  RESPONSE) 

1 8-  ME ASURE*  11.20583  (MOST  RECENT  RESPONSE  SPEED) 

1 9-  MEASURE=  289  294  7.  (TOTAL  INTEGRATIVE  ACTI VI TY=TI A) 

20- MEASURE*  193  202  7.  (TOTAL  FORWARD  INTEGRATIVE  ACTIVITY=TFIA> 

21 - MEASURE*  68  71  7.  (INTEGRATIONS  ACROSS  CATEGORIES) 

22- MEASURE*  23  23  7.  (INTEGRATIONS  WITHIN  CATEGORIES) 

23- MEASURE*  2 . 833333  ( PROP . CAT . I NTEG . ) 

24- MEASURE*  1.45679  (EMERGENCY  SHIFT  1) 

25- MEASURE*  2.144425  (EMERGENCY  SHIFT  2) 

26- MEASURE*  .2542373  (RECOVERY  1) 

27- MEASURE*  28  28  7.  (#  INFO  SEARCH  DEC.  ) 

28- MEASURE*  9  8  7.  <#  INFO  SEARCH  DEC.  I  NT.  ) 

29- MEASURE*  2.599999  (DISASTER  RESP. SPEED) 

30- MEASURE*  11.48461  (AVERAGE  DISASTER  RESPONSE  SPEED) 

31 - MEASURE*  31  (INTEGRATED  DISASTER  RESPONSE) 

32- MEASURE*  15  (APPLIED  DISASTER  RESPONSE) 

33- MEASURE*  13  (#  OF  DISASTER  DECISIONS) 

34- MEASURE*  8  (DISASTER  BACKWARD  INTEGRATIONS) 

35- MEASURE*  26  (DISASTER  TIA) 

36- MEASURE*  91  95  7.  (ADJUSTED  #  OF  FWD.  INTEG.  ) 

37- MEASURE*  .2542373  (RECOVERY  2) 


v.v.  .*.v 
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